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Monies granted unier Title VI-B of the Education of 
the Handicapped Act funded an accountability study of the Duval 
County Prograsi (Jacksonville, Florida) for trainable sentally 
handicapped (TRH) children and youth. An organized collection of 
approximately 675 behavioral objectives and accospanying criterion 
test itens vas developed for evaluating THR student perforsance and 
overall prograo effectiveness. Tfce objectives were incorporated in a 
voluae of 869 instructional objectives (A Catalog of Instructional 
Objectives for Trainable Mentally Betarded Students) in the areas or 
language developaent, social adequacy, and vocational readiness. The 
behavioral objectives and criterion test items were field tested by 
"teachers of the TBH, reviewed by experts, and compared with lajor 
cnrricnluu guides for THH prograas. Field testing indicated that the 
use of criterion test iteas with TMH students vas an expensive, 
tiae-consutiing, and disruptive evaluation technique. In an attempt to 
establish a more functional and less expensive evaluation 
alternative, a checklist of 80 sample behavior items vas compiled on 
the basis of importance ranking of skill areas by parents, teachers, 
and community agency personnel. Findings shoved that the checklist 
instrument vas highly reliable in and of itself and correlated very 
highly vith the use of corresponding criterion test items as an 
evaluation alternative. (GB) 
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INTRODUCTION 



The purpose of this rej»rt i% to provi<te a sxm^ty of ^he 
objecrives, activities, and field-test results for the thi -d and 
final year of an "Accountability Studly of t^e Program for Trainable 
Mentally iU;);arded Children and Youth/* 

The Accountability Study has been carried out by the Duval 
;otnty School Boards Jacksonville* Florida. Financial iup|X)rt 
und^r Title VI -B of the Education of the Handicapped Act was awarded 
the Puval County School Bcsrd oy the Bureau of Education for Ex- 
ceptional Students. Florida State Department of Education* 

For additional information concerning this project* the reader 
is referred to the Clearinghouse/ Information Center. Bureau of 
Education for Exceptional Stwtents. 319 Knot<^ t.ilding. Tallahassee. 
Florida 32304. 



BACKGROUND 



In Harch» 1971* the Duval County School Board submitted a proposal for 
federal assistance under Title VI-B of the Education of the Handicapped 
Act to conduct an Account«*iHtjr Stu4y of the ftjval County Program for 
Trainable Mentally Retarded Children and Youth. 

The process of developing an accountabilit;y system for evaluating the 
program for trainable mentally retarded (TMR) students required that 
specific skills appropriate for this population be identified, organized* 
and presented in a w^y that allowed student success or failure to be 
measured. An organized collection of behavioral objectives for TWl 
students was developed as the basis for evaluating student performance 
and overall program effectiveness. For each objective an accompanying 
criterion test item was written. These criterion test items sp'^cified 
the situation* materials* and directions to be used in formally assess* 
ing student performance on a given objective. By using these criterion 
test items as both pre and post tests* a measure of student progress could 
be obtained and the effectiverass of specific instructional programs 
determined. 

During the first two years of this accountability stu4y* teachers in the 
Duval County TMR program developed and field- tested a collection of 
approximately 550 behavioral objectives and corresponding criterion test 

2 



ERIC 



Items. These objectives covered skills in the areas of Social, Aca<Jemic, 
and ucc'pational Competence and were arranged by content into a modular 
organizational radel recommended by Drs. Louis Schwartz and Andrew 
Oseroff* Florida State University, and Or. Charles Forgnone, University 
of Florida. 

In an attempt to identify specific highly important objectives which 
could be used to sample student ability in a given skill area, each of 
the objectives and major skill areas in the organizational mode-1 was 
ranked by 122 teachers and 34 representatives of community agencies to 
determine the differential importance of the various skills. In addition, 
the 26 jaajor skill areas were importance- ranked by 97 parents of TKR 
students in the Duval County program, (See Sunmry Evaluation Report 1972- 
1973 J 

In order to develop a relatively comprehensive catalog of instructional 
objectives for THR students, skills that had been CHnitted thrjough over- 
sight from the collection of 550 objectives needed to be identified. 
Three procedures were used to identify omitted skills: (1) the collection 
of objectives was reviewed and critiqued by 122 teachers in TMR programs 
throughout the State of Florida; (2) a project consultant compared the 
objectives in the collection with the content of over fifteen curriculum 
guides for TMR students; and (3) a panel of three experts in the field 
of educating mentally retarded students conducted a thorough review of 
the collection of objectives. These three activities resulted in the 
identification of api^roximately 125 new objectives which needed to be 
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Included In the final catalog of objectiyes to be developed during the 
1973-74 project year. 



At the coevletion of the second project year» two major Usks remained 
in the developrent of a viable accountability system: (1} the omitted 
skills that had been identified needed to corporated into a final, 
relatively c(»nprehensi^ catalog of objecti js, and (2) the results of 
the 1972-73 field-test activities had to be evaluated; and the efficiency 
of criterion test items in assessing student performance had to be 
determined. 

PROJECT OBJECTIVES 

The overall stated objective for the 1973-74 project year i^as: 

"to solidify an accountability model with adequate 
documentation sc that it can be utilized by other 
programs to reliably evaluate behavioral change in 
the trainable mentally retarded student particlr 
pants." 

To complete the development of an accountability model two major categories 
of specific objectives were identified for ^.he l97i-74 project year. The 
first group of objectives focussed on revising and exparjing the 1972-73 
collection of objectives into a relatively ccn?)rehens1ve catalog of in- 
structional objectives for im students end on developing a collection 
of instructional classroom-ac^ vities that were rulated to the skills Iden- 
tified In the c^-itdlog. The secoud cluster of objectives for the 1973-74 
project year were related to the development And f ie^ i-testing of an 
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assessment alternative to criterion test item. The need for such an 
alternative was recognized when the results of the 1972-73 assessment 
activities Indicated that criterion test items were an inefficient and 
disruptive means of evaluating the f^rformance of TNR students. 

A) EXPAf^SION AND REVISION OBJECTIVES 

* To identify sheltered workshop skills* 

* To write objectives and criterion test items for sheltered 
workshop skills. 

* To develop objectives and criterion test items for the skills 
identified by the results of the validation techniques employed 
during the 1972-73^project year, 

* To provide a greater number of instructional alternatives for 
meeting the objectives by expanding the Instructional Materials, 
ftethOd-Mediat and Activities section of the model* 

* To revise the objectives developed during the first two project 
years in the areas of sociaU academic^ and occupational com- 
petence as indicated by the 1972-73 field-test. 

* To prepare introductions to both the catalog of objectives and 
the volume nf suggested instructional materials, resources, and 
activities. 

* To prepare a comprehensive accountability report on changes of 
behavior in tr>^inable mentally retaroed students. 

* To provide for printing and disseminating copies of project 
reports, the catalog of obj,^tives, and the voltaro of instruc- 
tional materials^ resources, and activities. 



B) ASSESSrCNT AITERNATIVE OBJECTIVES 



To assist the State Assessment and State Research and Development 

programs in transferring the accountability model into a viable 
state assessment mechenism. 

To develop an evaluation alternative to criterion test items for 
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assessing student status and progress on program objectl^s. 

* To compare the overall efficiency of this evaluation alterna- 
tive with the use of criterion test items* 

* To determine the reliability of this new evaluation alternative. 

PROJECT ACTIVITIES 

V 

A) EXPANSION AND REVISION ACTIVITIES 

In attempting to identify objectives appropriate for successful function- 
ing in sheltered workshop situations* th^. project director visited seven 
recomnended sheltered workshops throughout the state of Florida- As a 
result of these visitations, approximately 85 skills which contribute to 
successful performance in sheltered workshop situations were identified. 
The project director and a task force of project consultants transformed 
these skills into written behavioral objectives with acconq>anying criterion 
test items. 

Durinc the first six months of the 1973-74 project year, the project 
director, Duval County TMR teachers ♦ and project consultants extensively 
revised and expanded the Sensory-Motor, Conmunl cation Skills, Craft 
Production, and Sewing and Mending components of 1972-72 :ollection of 
objectives. These modifications were based on the recoimendations of 
Duval County TMR teachers who had field-tested the entire 1972-73 collection 
of objectives. During this san« period, behavioral objectives and criterion 
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test it«ns were also written for those skills identified through teacher 
and expert review of the 1972-73 rwterials and through a comparison of 
these materials with major curriculum guides for IVR programs. 

All of these new and revised objectives were then incorporated into the 
organizational model resulting in a volume of 869 instructional objectives* 
A Catalog of Instructional Objectives for Trainable Mentally Retarded 
Students > 

During the Sumner of 1973, five Oifval County TfS teachers worked for eight 
weeks identifying instructional activities* materials* and resources which 
would aid fellow teachers develop alternatives for instructing students in 
a variety of skill areas. These suggested activities* materials* and 
resources were expanded and revised by a project consultant during the 
course of the 1973-74 project year. Though extensive editing remains to 
be done on these materials* it is hoped that this collection of instructional 
suggestions will be available to teachers by September^ 1975. 

In early May* 1974 a project evaluation panel consisting of Dr. Harold H, 
Heller* University of Alabama; Dr. Oliver P. Kolstoe* University of Northern 
Colorado; and Dr. Bobby Palk* University of Alabama* reviewed the content 
of the 1973-74 catalog of objectives and the instructional alternatives 
developed by teachers, (This panel had evaluated the 1972-73 project 
materials and was thus able to adjudge the qualitative and quantitative 
differences in project products that had taken place during the period 
from May* 1973 to May, 1974.) Their report, based on a five day examination 
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of 1973-74 project roaterials, affirmed the attainment of the expansion and 
revision objectives of the project and concluded that the catalog of ob- 
jectives tnight becwne "a major resource to persons concerned with Improv- 
ing educational programs for TWi children". 

B) ASSESSMENT ALTERNATIVE ACTIVITIES 

As a result of field-testing the 1972-73 collection of objectives and 
accompanying criterion test items in Duval County, it was discovered that 
the use of criterion test items with im students was an expensive* time- 
consuming, and disruptive evaluation technique. Since groun-administered 
"paper and pencil" assessment Is extremely difficult* 1# not Impossible, 
with this population* each test Item required Individual administration. 
Substitute teachers were employed to Insure the maintenance of cWssroom 
management and the continuation of instructional programming while the 
regular filassroom teacter administered criterion test Items to individual 
students. To maximize the opportunity to maxe objective comparisons across 
classes, consultants were hired to develop uniform assessment materials 
for each criterion test item. It was also necessary to employ a clerk to 
manage these materials and to disseminate* collect, code, and tabulate the 
criterion test Items* In short, the use of criterion test Items did not 
prove to be a practical means of e^^aluatlng the performance/progress of 
TKR students even though such Instruments did possess the Important con- 
struct validity desired* 

It was, therefore, decided to establish a more easily used and less ex- 
pensive evaluation alternative. A checklist consisting of ^ Items 
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sampling behaviors in 19 cajor skill areas mss developed (See Appendix 
A). The skill areas to be sao^led were determined by asking approximately 
250 parents, teachers, and comnunity agency personnel to conduct an impor- 
tance-ranking of the 26 skill areas in the 1972-73 collection of objectives 
for TMR students. Once a hierarchy of skill area importance had been es- 
tablished, specific sample objectives from each skill area were selected 
from an importance-ranking of objectives perforred by 122 TMR teachers, 
i^®^ Summary Evaluation Report 1972-73.1 The most important objectives 
in each of the 19 skill areas as determined by this group of teachers were 
incorporated into the checklist. The resultant TfSl Assessment-Profile 
represents a behavior sample composed of the most important objectives in 
J»e most important skill areas as determined by parent, teacher, and 
community agency personnel ranking. 

Though the 80 item TMR Assessment-Profile possessed consensual validity 
based on the procedures used to determine its content, the overall reli- 
ability of this checklist and the concurrent validly of this instrument 
with the corre3pon<iing criterion test Items still remained to be detemined. 
The establishment of checkH.. reliability and the determination of 
checklist - criterion test item concurrent validity became major field-test 
activities in attempting to achieve the project objectives relating to 
the development of a viable assessment alternative to criterion test items. 
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FVALUATION OESION 

Formal evaluation required statistical analysis to determine reliability 
and validity of the Tm Assessment-Profile- Test-retest reliability was 
evaluated on one sample of 113 students, and inter-rater reliability was 
established on a separate sample of 109 students. It was not enough to 
look only at the reliability of the total checklist since possibilities 
exist for shortening or otherwise nwdifying this checklist in the future. 
Therefore, test-retest and inter^rater reliability information were ob- 
tained tor both the overall checklist and for the individual checklist 
items. 

Comparability of the checklist and its corresponding criterion test items 
(See Appendix B) was evaluated by administering both instrunents to a 
sample of 100 Duval County TMR students. The internal validity anu 
dimensionality of both instruments were evaluated by factor analyses. 

Finally, norms based upon the total of both the reliability and concurrent 
validity samples were established in terras of means and standard deviations 
as well as frequency distributions. To assist in future revisions of the 
checklist, the mean Mental Age and frequency distributions by Mental Age 
were obtained for students passing each diecklist item. 

The following chart illustrates the teacher - student samples used in the 
1973-74 TMR Asses'.ment-Profile field- test activities. 
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SAMPLES EMPLOYED 



EVALUAYIOM 

ODTirr m tor 


rrtt HttTv 
vUuri 1 T 


NO, OF 

DliOt 1 C 


NO. m 


Test- Re test 


4 outside counties 


1 1 J 




Inter- Rater 


Duval 


109 


27 


Instrument 
Comparison 


Duval 


ICO 


31 


Norms 


5 counties 


328 


60 


MA Distribution 


5 counties 


396 


63 



SUW1ARY OF FINDINGS 

Checklist test-retest reliability was determined by correlating the scores 
assigned 113 students the first time and the scores given the san^e subjects 
by their teachers a week later* The obtained product-movement correlation 
of .92 attests to the basic reliability of the process of rating the subjects. 
Agreement between different raters {with different standards and different 
knowledge about the subjects) could be expected to be much lower « For 109 
paired ratings the correlation of checklist inter-rater agreenient was a 
surprisingly high .84» al;;<4;st as high as the test-retest correlation. 

Individual checklist item reliability computed on both inter-rater and 
test-retest bases was high for most items. Most ranged from .40 to .79. 
Only two or three items were suspiciously low on reliability. These same 
items tended to show rather low correlations with their corresponding 
criterion test items. Overall comparison beti^en total checklist scor^ 
and the total score on the criterion test items was examined. The product- 
moment correlation was .80. This correlation indicates that the checklist 
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MS evaluating almost the sane :ittnbute of ability that the criterion test 
Items were* 

The question as to whether the two instruments were getting at similar 
abilities was examined further through factor analysis* By looking at the 
first principal axis* a type of item analysis, was available. The justifica- 
tion for using a summative total score is seen in that between 41% and 45v 
of the conwwn variance was contained in the first principal axis for these 
instruments. All checklist items had validity correlations with total 
checklist scores of greater than .30; most were above .50- Individual 
criterion test item did not have this same uniformly high correlation with 
the total score for all criterion test items. Some individual item - total 
criterion test item score correlations were near zero. However, nost were 
fairly large as they were for the checklist item - total checklist score 
coefficients. 

The Varimax rotated factors for checklist and criterion test items were 
rather different. This would indicate that while superficially the two 
instruronts are similar, there are some basic differences in what they 
sample. As long as total scores are being focussed on» they are very 
comparable. 

Since data on student and teacher variables were available, they were 
correlated with both criterion test item scores and checklist scores. As 
would be expected the pupil variables that correlated with both checklist 
and criterion test item scores were age and I.Q. The teacher variable 
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that correlated with both of these scores was grade or level taught. (As 
mighu Iv expectedt older students received higher scores.) The data ob- 
tained were examined for distributions* and norm tables were assembled. 
Since the Florida State Department of Education my split the checklist 
into three age-relevant sublists, a further analysis was done. Students 
passing each checklist item (score = 3) were sorted by Mental Age into one- 
year Intervals to illustrate the difficulty of each item. Mean and stan- 
dard cteviations appeared to provide an index of difficulty. There did 
not appear to be a very wide range of difficulty for the i terns • The 
range from highest to lowest mean Mental Age for students passing indivi* 
'jual items was only about 1 1/4 years. 

DISCUSSION OF FINDINGS 

As a result of 1973-74 field- test activities, it was determined that the 
correlation between assessment of student performance using the checklist 
fonnat of the 80 item TMR Assessment-Profile correlated at .80 with assess- 
ment of student performance using the corresponding &i criterion test items. 
In addition* a test-retest reliability of .92 and an inter-rater reliability 
of .84 ^ere established for the THR Assessment-Profile. On the basis of 
these findings^ it appears that this checklist instrument is highly reliable 
in and of itself and correlates very well with the use of corresponding 
criterion test items as an evaluation alternative. 

It should also be noted that the checklist format when con^ared with the 
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criterion test Item aU^rnatlve was <sis1er for teache:*? to use, d;<l not 
disrupt the ongoing instructional prograr^ and provided a mi^ condensed 
though equally expansive profile of student abilUy. While iz took two 
school oays to evaluate some students u^ln.^ the 80 '. Mterion test 1 terns » 
it took only from fl^^e to ten Tiinjtes to cofnp1e*e a checklist on the same 
student* This time far.tor when cou^*' with the c.ifference in the cost 
of materials for these two as^?5?ment alter/ietives makes ♦he checklist by 
far the more efficient instruront. ^Uen one considers tiiese factors of 
usability and efficiency ccnbined with the high reliability of the TMR 
^^us^ssment-Frofile and the strong correlation of this checklist mstrwnent 
with its criterion test it&^^ alternative, it appears tf^t c^ie checklist 
format of the TMR Asseisment-Prot'i is a hignly desirable and effective 
instrument for evaluating the performance of TMR stude'^ts. 
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Appendix A 
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Directions for Con^leting TW? Assessment-Profile 



This form is being used to gather infonnation on the student's abilities 
on 80 selected skills. Your observations of the student since the beginning 
of the school year are the basU for your judgment of the student's 
abilities. 

DIRECTIONS: 

"YES" Place a check (X) in this column if the student is able 

to correctly tegin and complete the skill upon request . 
(E.g* #1. ftequestl "Count f n»n 1 to 5." 
Student responds: "1, 2, 3, 4, 5." 
E^g. #2* Request; "Shampoo your hair.** 
Student responds by turning on watert wetting hair, 
applying shampoo, washing, rinsing, and drying hair, 
turning off water, and capping the shampoo bottle.) 

Checking "YES" means that the student will have little or no difficulty 
performing the skill upon request . 

"PARTIALLY** Place a {X) in this column if the student is able to 
perform part of the skill u^n request , 
{E.g. #1, Request: "Count from 1 to 5." 
Student responds: "1, 2, 3, 4." 
E.g. #2, Request: "Shampoo your hair.** 
Stuctent responds by turning on water.) 

Checking "PARTIALLY" means that the student can do part (large or little) 
of the skill. However, he is not able to perform the entire skill upon 
request. 

W Place a (X) in this column if the student is not able 

to perform any part of the skill upon request 
CF,g. #1, Itequest: "Count from 1 to 5." 
Stuient is non-verbal or gives no response. 
E.g. #2, Request: "Shampoo your hair." 
Stident makes no response which is needed to shampoo 
hair.) 

Checking "NO" means that the skill is teyond the student's abilities or 
that he will be unable to perforis any part of it upon request . 
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TMR ASSESSMENT - PROFILE 



STUDEKT: 

NO PARTIALLY 



TEACHER: 



YES 



CLEANLINESS 

1. Washes, dries hands 

2. Hashes, dries face 

3. Brushes teeth 

4. Wipes food from face 

5. Wipes food from hands 

EATING 

6. Picks up. chews, swallows solid food 

7. Drinks from glass 
8* Uses spoon 

9. Uses fork 

10. Uses knife 

SAFETY 

11. Knows dangerous objects 

12. Knows objects fiarmful if Ingested 

13. Knows objects hamful to eyes 

14. Knows objects harmful to ears 

15. Knows objects harmful to nose 

DRESSING AND IfflQRESSING 

16. Uses zipper (on person) 

17. Uses buttons (on person) 

18. Uses snaps (on person) 

19. Puts on shoes 

20. Puts on socks 

HEALTH 

21. Urinates appropriately 
GROOMING 

22. Combs hair 

23. Brushes hair 

24. Applies deodorant 

25. Knows own clothing 

26. Knows clothes appropriate for t^ather conditions 
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PARTIALLY YES 

SELF KNOWLEDGE 



27. States whole name 

28. States street address 

29. States telephone number 

30. Kn(MS own sex 

SOCIAL INTFRACTIOK 



31. Interacts positively In physical actions 

32. Interacts positively verbally 

33. Follows rules in oroup situations 

34. Works cooperatively on tasks 

35. ftespects, cares for property 

POSITION IN SPACE 



36* Recognizes iiiajor body parts 

37. Names major body parts 

38. Understands concepts: up-down 

39. Understands concepts: around*through 

40. Understands concepts: front^back 

MOTOR COORDINATION 

41. Grasps » picks up an object 

42. Climbs stairs 

43. Steps accurately 

44. Throws uncterhand 

45. Runs 

46. Opens, closes doors 
RECREATION AND SPORTS 

47. Behaves safely on playground 
LISTENING SKILLS 

48. Understands concepts: loud-soft 
SPOKEN LANGUAGE 

49. Repeats ten food words after teacher 

50. Independently produces ten food words 

51. (tecognlzes ten food words (spoken) 

52. Comprehends ten food words (spoken) 
53* Repeats three word phrase 

54. Dials private dial telephone 
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teRITTEN LANGUAGE 



55. Writes whole name 

56. Write* street address 

57. Writes phone number 

READING 

58. Moines ten fUshcard safety words 

59. Cwnprehends te^ safety words (written) 
60- Names ten flashcard public sign words 

6K Names ten flashcanl public buildind title words 

62. CofT^reheiids ten public sign words (written) 

COUNnNS AND NUMERAL RECOSNIHON 

63. Jcitfits orally (1-10) 

64. Coints objects (1-10) 

65. Nan«s flashcard numbers (0 - 10) 

66. Understands concepts: more-less 

67. Selects designated nur^r from qroup 

HONEY CONCEPTS 

68. Recognizes a>ins 

69. Names coins 

70. Reads prices under $1.(X) 
7K Exchanges money U - lOt 

HOUSEKEEPING 

72. Picks things up, puts them away 

73. Emptier trash 

74. Sweeps floor 

75. Cleans sink 

HOtgHAKlNG 

761 Puts awa^ Ironed clothes 

7/. Prepares water, washes, rinses dishes 

78. Opens bottles 

79. Opens jars 

80. Hangs up clothes on clothesline 
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CRITERION TEST ITEM 



WASHING AND DRYING 



(This criterion test item corresponds to item #1 on the THR Assessn^nt* 
Profile.) 



SUBJECT: 



(first) (initial! 



TEACHER: 



To test this objective you will need a sink^i water, soap^ and a cloth 
towel. Measure this objective when the student's hands are dirty. Tell 
the student to wash his hands with soap and water and dry them. You may 
give the instructions twice but do not prompt additionally either verbally 
after the student begins or by demonstration at any time. 



1. Turns on water 



6. Rinses off <*11 soap 



2. Applies water to hands 



7. Turns off water 



3. Applies soap 



8. Picks up towel 



4, Works soap over entire 
area of hands 



9. Dries hands with 
towel 



5* Rubs hands together 



10. Both hands clean 



6* Rinses soap off hands 



n. Both hands dry 



Total Score 
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